Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler 3.0 #965

Merged
merged 67 commits into from
Dec 9, 2019
Merged

Compiler 3.0 #965

merged 67 commits into from
Dec 9, 2019

Conversation

mohamed82008
Copy link
Member

@mohamed82008 mohamed82008 commented Nov 17, 2019

This WIP PR will include yet another refactor of the internals of Turing with the following main goals:

  1. Readability,
  2. Modularity to be able to move the compiler to DynamicPPL at some point,
  3. Flexibility and extensibility to enable the adding of features which are now tricky to implement, e.g. querying the likelihood, and finally
  4. Interoperability with other PPL approaches to enable interop with Soss, Gen and ProbabilityModels.

This PR currently implements the following features:

  • Full-fledged support for partially missing values with automatic differentiation. For example, the following will treat x[1] and x[3] as random variables and x[2] and x[4] as observations:
@model f(x) = begin
    for i in 1:length(x)
        x[i] ~ Normal()
    end
end
sample(f([missing, 1.0, missing, 2.0]), NUTS(10, 0.65), 1000)
  • The way in which arguments to the model are handled was significantly changed. Every argument now needs to be defined either when constructing the model or by setting a default value that works the same way a default value works in Julia. When an input variable can be missing, its input value should be missing. This can be input when constructing the model or using the default value. If this missing variable is a vector and we need to loop or broadcast over it inside the model, the user needs to do:
@model f(x, ::Type{T} = Float64) where {T} = begin
    if ismissing(x)
        x = Vector{T}(undef, 10)
    end
    for i in 1:length(x)
        x[i] ~ Normal()
    end
end

Note that any model argument whose value is not missing is not going to be handled as a random variable and will be treated as an observation. If an argument has no default value and is not input when constructing the Turing.Model, an error will be thrown. Note that the above ismissing if-block is only needed if we need to loop over x. If we only need x ~ MvNormal(...) or something similar, then there is no need to initialize it when ismissing(x) is true.

  • Running the model now happens in a context, DefaultContext by default to calculate the log joint probability. But we can also run the model in different contexts to evaluate variants of the logp. For example, I defined the LikelihoodContext which enables the computation of the log likelihood only instead of the log joint probability. Another context defined is the BatchContext that computes logprior + s * loglikelihood, where s = npoints / batch_size. This is important for stochastic (batch-based) gradient algorithms to be optimizing the original objective in the expectation, and not highly weighing the prior.
  • The user can now tap into the VarInfo object in the model definition using the macro @varinfo(). This is not a real macro and will throw if called outside the @model definition. However, inside the @model definition, one can use this to get a handle to the VarInfo object inside the model run for more advanced use cases. There is also @logpdf() which simply returns the current accumulated value of the log joint probability or whatever the context is having the model compute.
  • Random variable names are now not baked into the Turing.Model type and can be passed by the user using the special distribution type NamedDist(dist, name). So x ~ NamedDist(dist, name) will have the random variable named name not x. name can be a string or a Turing.VarName.
  • The VarName constructor is now very neat and intuitive. You can do vn = Turing.@varname x[1,:][i] to get a VarName{:x} with vn.indexing == "[1,Colon()][$i]", whatever i is.
  • And last but not least, broadcasting the ~ symbol is now supported for both observations and random variables. One can do x .~ dist to mean that all elements of x are distributed according to dist. The main difference between ~ and .~ in usage is that ~ doesn't assume the left-hand-side to be defined while .~ does. So using x .~ dist when x wasn't previously defined will lead to an error. Broadcasting will work column-wise when dist isa MultivariateDistribution and x isa AbstractMatrix but not AbstractMatrix{>:AbstractVector}. If x isa AbstractMatrix{Any} for example, the broadcasting semantics here become ambiguous so we throw an error. This should not be an issue for most use cases.

I still need to do more tests and rebase but I thought I should write this summary now because there will be no more features added to this PR, just fixes from now on.

@xukai92
Copy link
Member

xukai92 commented Nov 18, 2019

I will review it tomorrow.

@mohamed82008
Copy link
Member Author

Fixes #754.

src/Turing.jl Outdated Show resolved Hide resolved
src/Turing.jl Outdated Show resolved Hide resolved
src/Turing.jl Show resolved Hide resolved
src/core/RandomVariables.jl Show resolved Hide resolved
src/core/compiler.jl Show resolved Hide resolved
src/core/compiler.jl Show resolved Hide resolved
src/core/compiler.jl Outdated Show resolved Hide resolved
src/core/compiler.jl Outdated Show resolved Hide resolved
@mohamed82008
Copy link
Member Author

Please don't review this PR now except for the fun of it. I will break it down a bit.

@cpfiffer
Copy link
Member

Could we add a macro @logpdf that just wraps around the @vi macro and only returns the log density?

@mohamed82008
Copy link
Member Author

Could we add a macro @logpdf that just wraps around the @vi macro and only returns the log density?

Sure, I can do that.

@mohamed82008
Copy link
Member Author

I wrote a summary of the features added in this PR. I will get to more tests and rebasing as soon as I can.

@mohamed82008
Copy link
Member Author

mohamed82008 commented Dec 8, 2019

So there seem to be some version conflicts.

@mohamed82008
Copy link
Member Author

The tests are running properly now thanks to @goedman 's latest CmdStan release, thanks Rob!

@yebai yebai merged commit 7c20720 into master Dec 9, 2019
@delete-merged-branch delete-merged-branch bot deleted the mt/compiler3.0 branch December 9, 2019 00:14
@yebai
Copy link
Member

yebai commented Dec 9, 2019

Many thanks, @mohamed82008!

@goedman
Copy link

goedman commented Dec 9, 2019

Mohamed, just to make sure, are you referring to CmdStan v5.3.0 or v5.3.1.

I believe the latter as currently BinDeps and Homebrew are not version-wise compatible.

@mohamed82008
Copy link
Member Author

5.3.1 yes

@goedman
Copy link

goedman commented Dec 10, 2019

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants